Recently, methods of performing object identification and the like making use of machine learning are being studied. As part of a family of machine learning methods, deep learning, which uses a neural network with several hidden layers between an input layer and an output layer, shows high performance in recognition.
And, the neural network using the deep learning generally learns through backpropagation using losses.
In order to perform such learning of a deep learning network, training data are needed in which tags, i.e., labels, are added to individual data points by labelers. Preparing these training data (i.e. classifying the data correctly) can be very labour-intensive, expensive and inconvenient, especially if a large amount of the training data is to be used and if the quality of the data pre-preparation is not consistently high. Conventional interactive labeling can be computationally expensive and fail to deliver good results.
Therefore, recently, auto-labeling which adds tags, i.e., labels, to a training image using a deep learning-based automatic labeling device is popular, and inspectors inspect auto-labeled training images to correct the tags or the labels.
However, in such a conventional method, a throughput of the inspectors is low as compared to the throughput of the automatic labeling device. Therefore, it takes a long time to generate true labels for entire training images. In order to improve the overall throughput, the number of the inspectors must be increased, resulting in an increase of the cost.
Also, it is difficult to acquire a large number of the inspectors skilled enough to keep up with the throughput of the automatic labeling device.